Motor control similarity between speakers saying "a souk" using inverse atlas tongue modeling.

Finite element models (FEM) of the tongue have facilitated speech studies through analysis of internal muscle forces indirectly derived from imaging data. In this work, we build a uniform hexahedral FEM of a tongue atlas constructed from magnetic resonance imaging data of a healthy population. The FEM is driven by inverse internal tongue tissue kinematics of speakers temporally aligned and deformed into the same atlas space, while performing the speech task "a souk" allowing muscle activation predictions. This work aims to investigate the commonalities in tongue motor strategies in the articulation of "a souk" predicted by the inverse tongue atlas model. Our findings report variability among five speakers for estimated muscle activations with a similarity index using a dynamic time warp function. Two speakers show similarity index > 0.9 and two others < 0.7 with respect to a reference speaker for most tongue muscles. The relative motion tracking error of the model is less than 2% which is promising for speech study applications.

Synthesizing Audio from Tongue Motion During Speech Using Tagged MRI Via Transformer.

Investigating the relationship between internal tissue point motion of the tongue and oropharyngeal muscle deformation measured from tagged MRI and intelligible speech can aid in advancing speech motor control theories and developing novel treatment methods for speech related-disorders. However, elucidating the relationship between these two sources of information is challenging, due in part to the disparity in data structure between spatiotemporal motion fields (i.e., 4D motion fields) and one-dimensional audio waveforms. In this work, we present an efficient encoder-decoder translation network for exploring the predictive information inherent in 4D motion fields via 2D spectrograms as a surrogate of the audio data. Specifically, our encoder is based on 3D convolutional spatial modeling and transformer-based temporal modeling. The extracted features are processed by an asymmetric 2D convolution decoder to generate spectrograms that correspond to 4D motion fields. Furthermore, we incorporate a generative adversarial training approach into our framework to further improve synthesis quality on our generated spectrograms. We experiment on 63 paired motion field sequences and speech waveforms, demonstrating that our framework enables the generation of clear audio waveforms from a sequence of motion fields. Thus, our framework has the potential to improve our understanding of the relationship between these two modalities and inform the development of treatments for speech disorders.

Attentive Continuous Generative Self-training for Unsupervised Domain Adaptive Medical Image Translation.

Self-training is an important class of unsupervised domain adaptation (UDA) approaches that are used to mitigate the problem of domain shift, when applying knowledge learned from a labeled source domain to unlabeled and heterogeneous target domains. While self-training-based UDA has shown considerable promise on discriminative tasks, including classification and segmentation, through reliable pseudo-label filtering based on the maximum softmax probability, there is a paucity of prior work on self-training-based UDA for generative tasks, including image modality translation. To fill this gap, in this work, we seek to develop a generative self-training (GST) framework for domain adaptive image translation with continuous value prediction and regression objectives. Specifically, we quantify both aleatoric and epistemic uncertainties within our GST using variational Bayes learning to measure the reliability of synthesized data. We also introduce a self-attention scheme that de-emphasizes the background region to prevent it from dominating the training process. The adaptation is then carried out by an alternating optimization scheme with target domain supervision that focuses attention on the regions with reliable pseudo-labels. We evaluated our framework on two cross-scanner/center, inter-subject translation tasks, including tagged-to-cine magnetic resonance (MR) image translation and T1-weighted MR-to-fractional anisotropy translation. Extensive validations with unpaired target domain data showed that our GST yielded superior synthesis performance in comparison to adversarial training UDA methods.

Attentive continuous generative self-training for unsupervised domain adaptive medical image translation.

Self-training is an important class of unsupervised domain adaptation (UDA) approaches that are used to mitigate the problem of domain shift, when applying knowledge learned from a labeled source domain to unlabeled and heterogeneous target domains. While self-training-based UDA has shown considerable promise on discriminative tasks, including classification and segmentation, through reliable pseudo-label filtering based on the maximum softmax probability, there is a paucity of prior work on self-training-based UDA for generative tasks, including image modality translation. To fill this gap, in this work, we seek to develop a generative self-training (GST) framework for domain adaptive image translation with continuous value prediction and regression objectives. Specifically, we quantify both aleatoric and epistemic uncertainties within our GST using variational Bayes learning to measure the reliability of synthesized data. We also introduce a self-attention scheme that de-emphasizes the background region to prevent it from dominating the training process. The adaptation is then carried out by an alternating optimization scheme with target domain supervision that focuses attention on the regions with reliable pseudo-labels. We evaluated our framework on two cross-scanner/center, inter-subject translation tasks, including tagged-to-cine magnetic resonance (MR) image translation and T1-weighted MR-to-fractional anisotropy translation. Extensive validations with unpaired target domain data showed that our GST yielded superior synthesis performance in comparison to adversarial training UDA methods.

Rainbow Colormaps Are Not All Bad.

Some 15 years ago, Visualization Viewpoints published an influential article titled Rainbow Color Map (Still) Considered Harmful (Borland and Taylor, 2007). The paper argued that the "rainbow colormap's characteristics of confusing the viewer, obscuring the data and actively misleading interpretation make it a poor choice for visualization." Subsequent articles often repeat and extend these arguments, so much so that avoiding rainbow colormaps, along with their derivatives, has become dogma in the visualization community. Despite this loud and persistent recommendation, scientists continue to use rainbow colormaps. Have we failed to communicate our message, or do rainbow colormaps offer advantages that have not been fully appreciated? We argue that rainbow colormaps have properties that are underappreciated by existing design conventions. We explore key critiques of the rainbow in the context of recent research to understand where and how rainbows might be misunderstood. Choosing a colormap is a complex task, and rainbow colormaps can be useful for selected applications.

Analysis of Tongue Muscle Strain During Speech From Multimodal Magnetic Resonance Imaging.

PURPOSE: Muscle groups within the tongue in healthy and diseased populations show different behaviors during speech. Visualizing and quantifying strain patterns of these muscle groups during tongue motion can provide insights into tongue motor control and adaptive behaviors of a patient. METHOD: We present a pipeline to estimate the strain along the muscle fiber directions in the deforming tongue during speech production. A deep convolutional network estimates the crossing muscle fiber directions in the tongue using diffusion-weighted magnetic resonance imaging (MRI) data acquired at rest. A phase-based registration algorithm is used to estimate motion of the tongue muscles from tagged MRI acquired during speech. After transforming both muscle fiber directions and motion fields into a common atlas space, strain tensors are computed and projected onto the muscle fiber directions, forming so-called strains in the line of actions (SLAs) throughout the tongue. SLAs are then averaged over individual muscles that have been manually labeled in the atlas space using high-resolution T2-weighted MRI. Data were acquired, and this pipeline was run on a cohort of eight healthy controls and two glossectomy patients. RESULTS: The crossing muscle fibers reconstructed by the deep network show orthogonal patterns. The strain analysis results demonstrate consistency of muscle behaviors among some healthy controls during speech production. The patients show irregular muscle patterns, and their tongue muscles tend to show more extension than the healthy controls. CONCLUSIONS: The study showed visual evidence of correlation between two muscle groups during speech production. Patients tend to have different strain patterns compared to the controls. Analysis of variations in muscle strains can potentially help develop treatment strategies in oral diseases. SUPPLEMENTAL MATERIAL: https://doi.org/10.23641/asha.21957011.

Speech Audio Synthesis from Tagged MRI and Non-Negative Matrix Factorization via Plastic Transformer.

The tongue's intricate 3D structure, comprising localized functional units, plays a crucial role in the production of speech. When measured using tagged MRI, these functional units exhibit cohesive displacements and derived quantities that facilitate the complex process of speech production. Non-negative matrix factorization-based approaches have been shown to estimate the functional units through motion features, yielding a set of building blocks and a corresponding weighting map. Investigating the link between weighting maps and speech acoustics can offer significant insights into the intricate process of speech production. To this end, in this work, we utilize two-dimensional spectrograms as a proxy representation, and develop an end-to-end deep learning framework for translating weighting maps to their corresponding audio waveforms. Our proposed plastic light transformer (PLT) framework is based on directional product relative position bias and single-level spatial pyramid pooling, thus enabling flexible processing of weighting maps with variable size to fixed-size spectrograms, without input information loss or dimension expansion. Additionally, our PLT framework efficiently models the global correlation of wide matrix input. To improve the realism of our generated spectrograms with relatively limited training samples, we apply pair-wise utterance consistency with Maximum Mean Discrepancy constraint and adversarial training. Experimental results on a dataset of 29 subjects speaking two utterances demonstrated that our framework is able to synthesize speech audio waveforms from weighting maps, outperforming conventional convolution and transformer models.

CMRI2SPEC: CINE MRI SEQUENCE TO SPECTROGRAM SYNTHESIS VIA A PAIRWISE HETEROGENEOUS TRANSLATOR.

Multimodal representation learning using visual movements from cine magnetic resonance imaging (MRI) and their acoustics has shown great potential to learn shared representation and to predict one modality from another. Here, we propose a new synthesis framework to translate from cine MRI sequences to spectrograms with a limited dataset size. Our framework hinges on a novel fully convolutional heterogeneous translator, with a 3D CNN encoder for efficient sequence encoding and a 2D transpose convolution decoder. In addition, a pairwise correlation of the samples with the same speech word is utilized with a latent space representation disentanglement scheme. Furthermore, an adversarial training approach with generative adversarial networks is incorporated to provide enhanced realism on our generated spectrograms. Our experimental results, carried out with a total of 63 cine MRI sequences alongside speech acoustics, show that our framework improves synthesis accuracy, compared with competing methods. Our framework thereby has shown the potential to aid in better understanding the relationship between the two modalities.

Structure-aware Unsupervised Tagged-to-Cine MRI Synthesis with Self Disentanglement.

Cycle reconstruction regularized adversarial training-e.g., CycleGAN, DiscoGAN, and DualGAN-has been widely used for image style transfer with unpaired training data. Several recent works, however, have shown that local distortions are frequent, and structural consistency cannot be guaranteed. Targeting this issue, prior works usually relied on additional segmentation or consistent feature extraction steps that are task-specific. To counter this, this work aims to learn a general add-on structural feature extractor, by explicitly enforcing the structural alignment between an input and its synthesized image. Specifically, we propose a novel input-output image patches self-training scheme to achieve a disentanglement of underlying anatomical structures and imaging modalities. The translator and structure encoder are updated, following an alternating training protocol. In addition, the information w.r.t. imaging modality can be eliminated with an asymmetric adversarial game. We train, validate, and test our network on 1,768, 416, and 1,560 unpaired subject-independent slices of tagged and cine magnetic resonance imaging from a total of twenty healthy subjects, respectively, demonstrating superior performance over competing methods.

Validation of Muscle Fiber Architecture of the Human Tongue Revealed by Diffusion Magnetic Resonance Imaging With Histology Verification.

PURPOSE: The goal of this study is to validate the muscle architecture derived from both ex vivo and in vivo diffusion-weighted magnetic resonance imaging (dMRI) of the human tongue with histology of an ex vivo tongue. METHOD: dMRI was acquired with a 200-direction high angular resolution diffusion imaging (HARDI) diffusion scheme for both a postmortem head (imaged within 48 hr after death) and a healthy volunteer. After MRI, the postmortem head was fixed and the tongue excised for hematoxylin and eosin (H&E) staining and histology imaging. Structure tensor images were generated from the stained images to better demonstrate muscle fiber orientations. The tongue muscle fiber orientations, estimated from dMRI, were visualized using the tractogram, a novel representation of crossing fiber orientations, and compared against the histology images of the ex vivo tongue. RESULTS: Muscle fibers identified in the tractograms showed good correspondence with those appearing in the histology images. We further demonstrated tongue muscle architecture in in vivo tractograms for the entire tongue. CONCLUSION: The study demonstrates that dMRI can accurately reveal the complex muscle architecture of the human tongue and may potentially benefit planning and evaluation of oral surgery and research on speech and swallowing.

Measuring Strain in Diffusion-Weighted Data Using Tagged Magnetic Resonance Imaging.

Accurate strain measurement in a deforming organ has been essential in motion analysis using medical images. In recent years, internal tissue's in vivo motion and strain computation has been mostly achieved through dynamic magnetic resonance (MR) imaging. However, such data lack information on tissue's intrinsic fiber directions, preventing computed strain tensors from being projected onto a direction of interest. Although diffusion-weighted MR imaging excels at providing fiber tractography, it yields static images unmatched with dynamic MR data. This work reports an algorithm workflow that estimates strain values in the diffusion MR space by matching corresponding tagged dynamic MR images. We focus on processing a dataset of various human tongue deformations in speech. The geometry of tongue muscle fibers is provided by diffusion tractography, while spatiotemporal motion fields are provided by tagged MR analysis. The tongue's deforming shapes are determined by segmenting a synthetic cine dynamic MR sequence generated from tagged data using a deep neural network. Estimated motion fields are transformed into the diffusion MR space using diffeomorphic registration, eventually leading to strain values computed in the direction of muscle fibers. The method was tested on 78 time volumes acquired during three sets of specific tongue deformations including both speech and protrusion motion. Strain in the line of action of seven internal tongue muscles was extracted and compared both intra- and inter-subject. Resulting compression and stretching patterns of individual muscles revealed the unique behavior of individual muscles and their potential activation pattern.

Tagged-MRI Sequence to Audio Synthesis via Self Residual Attention Guided Heterogeneous Translator.

Understanding the underlying relationship between tongue and oropharyngeal muscle deformation seen in tagged-MRI and intelligible speech plays an important role in advancing speech motor control theories and treatment of speech related-disorders. Because of their heterogeneous representations, however, direct mapping between the two modalities-i.e., two-dimensional (mid-sagittal slice) plus time tagged-MRI sequence and its corresponding one-dimensional waveform-is not straightforward. Instead, we resort to two-dimensional spectrograms as an intermediate representation, which contains both pitch and resonance, from which to develop an end-to-end deep learning framework to translate from a sequence of tagged-MRI to its corresponding audio waveform with limited dataset size. Our framework is based on a novel fully convolutional asymmetry translator with guidance of a self residual attention strategy to specifically exploit the moving muscular structures during speech. In addition, we leverage a pairwise correlation of the samples with the same utterances with a latent space representation disentanglement strategy. Furthermore, we incorporate an adversarial training approach with generative adversarial networks to offer improved realism on our generated spectrograms. Our experimental results, carried out with a total of 63 tagged-MRI sequences alongside speech acoustics, showed that our framework enabled the generation of clear audio waveforms from a sequence of tagged-MRI, surpassing competing methods. Thus, our framework provides the great potential to help better understand the relationship between the two modalities.

Generative Self-training for Cross-domain Unsupervised Tagged-to-Cine MRI Synthesis.

Self-training based unsupervised domain adaptation (UDA) has shown great potential to address the problem of domain shift, when applying a trained deep learning model in a source domain to unlabeled target domains. However, while the self-training UDA has demonstrated its effectiveness on discriminative tasks, such as classification and segmentation, via the reliable pseudo-label selection based on the softmax discrete histogram, the self-training UDA for generative tasks, such as image synthesis, is not fully investigated. In this work, we propose a novel generative self-training (GST) UDA framework with continuous value prediction and regression objective for cross-domain image synthesis. Specifically, we propose to filter the pseudo-label with an uncertainty mask, and quantify the predictive confidence of generated images with practical variational Bayes learning. The fast test-time adaptation is achieved by a round-based alternative optimization scheme. We validated our framework on the tagged-to-cine magnetic resonance imaging (MRI) synthesis problem, where datasets in the source and target domains were acquired from different scanners or centers. Extensive validations were carried out to verify our framework against popular adversarial training UDA methods. Results show that our GST, with tagged MRI of test subjects in new target domains, improved the synthesis quality by a large margin, compared with the adversarial training UDA methods.

DUAL-CYCLE CONSTRAINED BIJECTIVE VAE-GAN FOR TAGGED-TO-CINE MAGNETIC RESONANCE IMAGE SYNTHESIS.

Tagged magnetic resonance imaging (MRI) is a widely used imaging technique for measuring tissue deformation in moving organs. Due to tagged MRI's intrinsic low anatomical resolution, another matching set of cine MRI with higher resolution is sometimes acquired in the same scanning session to facilitate tissue segmentation, thus adding extra time and cost. To mitigate this, in this work, we propose a novel dual-cycle constrained bijective VAE-GAN approach to carry out tagged-to-cine MR image synthesis. Our method is based on a variational autoencoder backbone with cycle reconstruction constrained adversarial training to yield accurate and realistic cine MR images given tagged MR images. Our framework has been trained, validated, and tested using 1,768, 416, and 1,560 subject-independent paired slices of tagged and cine MRI from twenty healthy subjects, respectively, demonstrating superior performance over the comparison methods. Our method can potentially be used to reduce the extra acquisition time and cost, while maintaining the same workflow for further motion analyses.

A deep joint sparse non-negative matrix factorization framework for identifying the common and subject-specific functional units of tongue motion during speech.

Intelligible speech is produced by creating varying internal local muscle groupings-i.e., functional units-that are generated in a systematic and coordinated manner. There are two major challenges in characterizing and analyzing functional units. First, due to the complex and convoluted nature of tongue structure and function, it is of great importance to develop a method that can accurately decode complex muscle coordination patterns during speech. Second, it is challenging to keep identified functional units across subjects comparable due to their substantial variability. In this work, to address these challenges, we develop a new deep learning framework to identify common and subject-specific functional units of tongue motion during speech. Our framework hinges on joint deep graph-regularized sparse non-negative matrix factorization (NMF) using motion quantities derived from displacements by tagged Magnetic Resonance Imaging. More specifically, we transform NMF with sparse and graph regularizations into modular architectures akin to deep neural networks by means of unfolding the Iterative Shrinkage-Thresholding Algorithm to learn interpretable building blocks and associated weighting map. We then apply spectral clustering to common and subject-specific weighting maps from which we jointly determine the common and subject-specific functional units. Experiments carried out with simulated datasets show that the proposed method achieved on par or better clustering performance over the comparison methods.Experiments carried out with in vivo tongue motion data show that the proposed method can determine the common and subject-specific functional units with increased interpretability and decreased size variability.

Floor-of-the-Mouth Muscle Function Analysis Using Dynamic Magnetic Resonance Imaging.

To advance our understanding of speech motor control, it is essential to image and assess dynamic functional patterns of internal structures caused by the complex muscle anatomy inside the human tongue. Speech pathologists are investigating into new tools that help assessment of internal tongue muscle's cooperative mechanics on top of their anatomical differences. Previous studies using dynamic magnetic resonance imaging (MRI) of the tongue revealed that tongue muscles tend to function in different groups during speech, especially the floor-of-the-mouth (FOM) muscles. In this work, we developed a method that analyzed the unique functional pattern of the FOM muscles in speech. First, four-dimensional motion fields of the whole tongue were computed using tagged MRI. Meanwhile, a statistical atlas of the tongue was constructed to form a common space for subject comparison, while a manually delineated mask of internal tongue muscles was used to separate individual muscle's motion. Then we computed four-dimensional motion correlation between each muscle and the FOM muscle group. Finally, dynamic correlation of different muscle groups was compared and evaluated. We used data from a study group of nineteen subjects including both healthy controls and oral cancer patients. Results revealed that most internal tongue muscles coordinated in a similar pattern in speech while the FOM muscles followed a unique pattern that helped supporting the tongue body and pivoting its rotation. The proposed method can help provide further interpretation of clinical observations and speech motor control from an imaging point of view.

Prospective motion detection and re-acquisition in diffusion MRI using a phase image-based method-Application to brain and tongue imaging.

PURPOSE: To develop an image-based motion-robust diffusion MRI (dMRI) acquisition framework that is able to minimize motion artifacts caused by rigid and nonrigid motion, applicable to both brain and tongue dMRI. METHODS: We developed a novel prospective motion-correction technique in dMRI using a phase image-based real-time motion-detection method (PITA-MDD) with re-acquisition of motion-corrupted images. The prospective PITA-MDD acquisition technique was tested in the brains and tongues of volunteers. The subjects were instructed to move their heads or swallow, to induce motion. Motion-detection efficacy was validated against visual inspection as the gold standard. The effect of the PITA-MDD technique on diffusion-parameter estimates was evaluated by comparing reconstructed fiber tracts using tractography with and without re-acquisition. RESULTS: The prospective PITA-MDD technique was able to effectively and accurately detect motion-corrupted data as compared with visual inspection. Tractography results demonstrated that PITA-MDD motion detection followed by re-acquisition helps in recovering lost and misshaped fiber tracts in the brain and tongue that would otherwise be corrupted by motion and yield erroneous estimates of the diffusion tensor. CONCLUSION: A prospective PITA-MDD technique was developed for dMRI acquisition, providing improved dMRI image quality and motion-robust diffusion estimation of the brain and tongue.

Operational Recommendations for Scarce Resource Allocation in a Public Health Crisis.

The coronavirus disease 2019 pandemic may require rationing of various medical resources if demand exceeds supply. Theoretical frameworks for resource allocation have provided much needed ethical guidance, but hospitals still need to address objective practicalities and legal vetting to operationalize scarce resource allocation schemata. To develop operational scarce resource allocation processes for public health catastrophes, including the coronavirus disease 2019 pandemic, five health systems in Maryland formed a consortium-with diverse expertise and representation-representing more than half of all hospitals in the state. Our efforts built on a prior statewide community engagement process that determined the values and moral reference points of citizens and health-care professionals regarding the allocation of ventilators during a public health catastrophe. Through a partnership of health systems, we developed a scarce resource allocation framework informed by citizens' values and by general expert consensus. Allocation schema for mechanical ventilators, ICU resources, blood components, novel therapeutics, extracorporeal membrane oxygenation, and renal replacement therapies were developed. Creating operational algorithms for each resource posed unique challenges; each resource's varying nature and underlying data on benefit prevented any single algorithm from being universally applicable. The development of scarce resource allocation processes must be iterative, legally vetted, and tested. We offer our processes to assist other regions that may be faced with the challenge of rationing health-care resources during public health catastrophes.

Identifying the Common and Subject-specific Functional Units of Speech Movements via a Joint Sparse Non-negative Matrix Factorization Framework.

The tongue is capable of producing intelligible speech because of successful orchestration of muscle groupings-i.e., functional units-of the highly complex muscles over time. Due to the different motions that tongues produce, functional units are transitional structures which transform muscle activity to surface tongue geometry and they vary significantly from one subject to another. In order to compare and contrast the location and size of functional units in the presence of such substantial inter-person variability, it is essential to study both common and subject-specific functional units in a group of people carrying out the same speech task. In this work, a new normalization technique is presented to simultaneously identify the common and subject-specific functional units defined in the tongue when tracked by tagged magnetic resonance imaging. To achieve our goal, a joint sparse non-negative matrix factorization framework is used, which learns a set of building blocks and subject-specific as well as common weighting matrices from motion quantities extracted from displacements. A spectral clustering technique is then applied to the subject-specific and common weighting matrices to determine the subject-specific functional units for each subject and the common functional units across subjects. Our experimental results using in vivo tongue motion data show that our approach is able to identify the common and subject-specific functional units with reduced size variability of tongue motion during speech.

Analysis of fiber strain in the human tongue during speech.

This study investigates mechanical cooperation among tongue muscles. Five volunteers were imaged using tagged magnetic resonance imaging to quantify spatiotemporal kinematics while speaking. Waveforms of strain in the line of action of fibers (SLAF) were estimated by projecting strain tensors onto a model of fiber directionality. SLAF waveforms were temporally aligned to determine consistency across subjects and correlation across muscles. The cohort exhibited consistent patterns of SLAF, and muscular extension-contraction was correlated. Volume-preserving tongue movement in speech generation can be achieved through multiple paths, but the study reveals similarities in motion patterns and muscular action-despite anatomical (and other) dissimilarities.